Project Sekai
🔒 GDG Algiers CTF 2022 / 🩸-misc-mlm
0.pkl to 201.pkl, loaded pickle, and noticed in each numpy array, numbers are either all close to 0 or all close to 1. Then if I use "0" to represent all close to 0, and "1" for all close to 1, I got a binary string
0001000000000100000100000000010000010000000001000001000000000100000100000000010000010000000001000001000000000100000100000000010000010000000001000001000000000100000100000000010000010000000001000001000010
But nothing seems related to flag. I am pretty sure it's correct, but I did not see flag inside the binary string. 
pks = []
for i in range(1, 404, 2):
file = f"session_{i}.pkl"
t = np.load(open(file, "rb"), allow_pickle=True)
pks.append(t) pkl files (layer0.pkl to layer201.pkl) and attached in above zip mlm.zippks which is a 2D array of numpy array, each numpy array is weights of a given layer (0 to 201)I play CTFs ****, output is weekly.sahuang — Today at 11:01 PM
Does it make sense to assume input is masked flag, output is the masked part(i.e. the text inside flag format?)
Aymen — Today at 11:02 PM
You're on the right track, try to read more about how MLM works and how you could use it to get the flagpks, 3) feed it flag format Cyber...{***(MASKED)***}, output is probably the word they wantsahuang — Today at 11:20 PM
Some layers provided have array dimension much larger than 768, which is BERT dimension per layer, I guess that's something I should sort out?
Aymen — Today at 11:22 PM
Yes220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
220-You are user number 2 of 5 allowed.
220-Local time is now 10:18. Server port: 21.
220-This is a private system - No anonymous login
220-IPv6 connections are also welcome on this server.
220 You will be disconnected after 15 minutes of inactivity.
USER alBERT
331 User alBERT OK. Password required
PASS dBASE
230 OK. Current directory is /
CWD .
250 OK. Current directory is /
TYPE I
200 TYPE is now 8-bit binary
PASV
227 Entering Passive Mode (127,0,0,1,117,49)
RETR layer153.pkl
150-Accepted data connection
150 2304.2 kbytes to download
226-File successfully transferred
226 0.001 seconds (measured here), 1529.63 Mbytes per second pkl files (layer0.pkl to layer201.pkl) and attached in above zip mlm.zippks which is a 2D array of numpy array, each numpy array is weights of a given layer (0 to 201)I play CTFs ****, output is weekly.sahuang — Today at 11:01 PM
Does it make sense to assume input is masked flag, output is the masked part(i.e. the text inside flag format?)
Aymen — Today at 11:02 PM
You're on the right track, try to read more about how MLM works and how you could use it to get the flagpks, 3) feed it flag format Cyber...{***(MASKED)***}, output is probably the word they wantsahuang — Today at 11:20 PM
Some layers provided have array dimension much larger than 768, which is BERT dimension per layer, I guess that's something I should sort out?
Aymen — Today at 11:22 PM
Yes 
Ouxs — Today at 7:40 PM
you need to load the layers into the model
and just like we stated in the last hint reshape must be done
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained("bert-base-uncased")
print(model.num_parameters)
so basically every parameter of model is given as a separate pickle file
and need to merge them to create bert model
<bound method ModuleUtilsMixin.num_parameters of BertModel(
(embeddings): BertEmbeddings(
(word_embeddings): Embedding(30522, 768, padding_idx=0)
(position_embeddings): Embedding(512, 768)
(token_type_embeddings): Embedding(2, 768)
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False)
first part of output is word_embeddings, 30522 * 768 = 23440896
and length of first pkl file is indeed that 
>>> from transformers import pipeline
>>> model = pipeline('fill-mask', model='bert-base-uncased')
>>> pred = model("What is [MASK] name?")
>>> pred
[{'score': 0.5362833738327026, 'token': 2115, 'token_str': 'your', 'sequence': 'what is your name?'}, {'score': 0.260379433631897, 'token': 2014, 'token_str': 'her', 'sequence': 'what is her name?'}, {'score': 0.14665310084819794, 'token': 2010, 'token_str': 'his', 'sequence': 'what is his name?'}, {'score': 0.036417704075574875, 'token': 2026, 'token_str': 'my', 'sequence': 'what is my name?'}, {'score': 0.004835808649659157, 'token': 2049, 'token_str': 'its', 'sequence': 'what is its name?'}] 
from transformers import BertTokenizer, BertForMaskedLM
from torch.nn import functional as F
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased', return_dict = True)
text = "The capital of France, " + tokenizer.mask_token + ", contains the Eiffel Tower."
input = tokenizer.encode_plus(text, return_tensors = "pt")
mask_index = torch.where(input["input_ids"][0] == tokenizer.mask_token_id)
output = model(**input)
logits = output.logits
softmax = F.softmax(logits, dim = -1)
mask_word = softmax[0, mask_index, :]
top_10 = torch.topk(mask_word, 10, dim = 1)[1][0]
for token in top_10:
word = tokenizer.decode([token])
new_sentence = text.replace(tokenizer.mask_token, word)
print(new_sentence)
to use input
from https://towardsdatascience.com/how-to-use-bert-from-the-hugging-face-transformer-library-d373a22b0209
from transformers import BertTokenizer, BertForMaskedLM
from torch.nn import functional as F
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased', return_dict = True)
text = "The capital of France, " + tokenizer.mask_token + ", contains the Eiffel Tower."
input = tokenizer.encode_plus(text, return_tensors = "pt")
mask_index = torch.where(input["input_ids"][0] == tokenizer.mask_token_id)
output = model(**input)
logits = output.logits
softmax = F.softmax(logits, dim = -1)
mask_word = softmax[0, mask_index, :]
top_10 = torch.topk(mask_word, 10, dim = 1)[1][0]
for token in top_10:
word = tokenizer.decode([token])
new_sentence = text.replace(tokenizer.mask_token, word)
print(new_sentence)
to use input
from https://towardsdatascience.com/how-to-use-bert-from-the-hugging-face-transformer-library-d373a22b0209 >>> from transformers import pipeline
>>> model = pipeline('fill-mask', model='bert-base-uncased')
>>> pred = model("What is [MASK] name?")
>>> pred
[{'score': 0.5362833738327026, 'token': 2115, 'token_str': 'your', 'sequence': 'what is your name?'}, {'score': 0.260379433631897, 'token': 2014, 'token_str': 'her', 'sequence': 'what is her name?'}, {'score': 0.14665310084819794, 'token': 2010, 'token_str': 'his', 'sequence': 'what is his name?'}, {'score': 0.036417704075574875, 'token': 2026, 'token_str': 'my', 'sequence': 'what is my name?'}, {'score': 0.004835808649659157, 'token': 2049, 'token_str': 'its', 'sequence': 'what is its name?'}] 
Well done, you got first blood!
torch.from_numpy(pks[j]).view(shapes[j]) 
1